chore(tracing): avoid reentrant SpanData borrow panic by juanjux · Pull Request #19699 · DataDog/dd-trace-py

juanjux · 2026-08-14T12:27:54Z

Description

Dogweb's dd-trace-py 4.14.0rc2 dogfooding exposed a re-entrant PyO3 borrow panic under CI Visibility, Flask, and SQLAlchemy workloads:

pyo3_runtime.PanicException: Already mutably borrowed: PyBorrowError

The failure starts when a non-primitive span tag is converted to a string. In the observed case, the object's __str__ accessed an expired SQLAlchemy attribute, which issued a query and started a nested database span. Starting that span calls context_provider.active().

SpanData._set_attribute previously held a mutable PyO3 borrow for the entire method, including while calling user-defined __str__ and __index__. The native context provider introduced in 4.14 reads the active SpanData with a shared borrow. Re-entering it during string conversion therefore attempted a shared borrow while the mutable borrow was still live, causing PyO3 to panic. When this occurred during a SQLAlchemy transaction, the abrupt exception could also leave the session in the prepared state and cause later tests to fail.

This change performs all potentially re-entrant Python coercion before borrowing SpanData mutably. The mutable borrow is now limited to updating the native attribute map, and replaced Python values are dropped only after that borrow is released. Bulk and default attribute setters follow the same pattern.

Additional Notes

The SQLAlchemy prepared session errors are downstream fallout from the panic, not a separate SQLAlchemy integration issue. This fixes the invalid borrow at its source rather than catching or suppressing PanicException in an integration.

Coerce Python span attributes before taking a mutable native borrow so lazy values can safely start nested spans.

datadog-prod-us1-3 · 2026-08-14T12:28:43Z

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 10 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64, 1]

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64]

View all 10 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 8905c27 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8e1f22327

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-08-14T12:39:58Z

This change is marked for backport to 4.14 and it does not conflict with that branch.
The command used to test backporting was

git fetch origin 4.14 && git checkout origin/4.14 && git checkout -b backport-19699-to-4.14 && git cherry-pick -x --mainline 1 34109749a48537b81e382a218c3e38aafd1848f6

pr-commenter · 2026-08-14T12:56:17Z

Benchmarks

Benchmark execution time: 2026-08-14 16:35:06

Comparing candidate commit 8905c27 in PR branch juanjux/fix-reentrant-span-data-borrow with baseline commit 3e902cc in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 8 performance regressions! Performance is the same for 610 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.441µs; +2.608µs] or [+11.227%; +11.994%]

scenario:iastaspects-repr_aspect

🟥 execution_time [+77.472µs; +83.002µs] or [+20.813%; +22.298%]

scenario:iastaspects-upper_aspect

🟥 execution_time [+53.078µs; +57.688µs] or [+22.536%; +24.494%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+130.254µs; +138.406µs] or [+32.162%; +34.175%]

scenario:iastaspectssplit-rsplit_aspect

🟥 execution_time [+11.599µs; +17.020µs] or [+8.093%; +11.875%]

scenario:span-start

🟥 execution_time [+1.359ms; +1.535ms] or [+8.878%; +10.027%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+522.396ns; +563.948ns] or [+19.762%; +21.334%]

scenario:tracer-small

🟥 execution_time [+30.444µs; +32.975µs] or [+9.008%; +9.758%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

unstable execution_time [-760.077ns; +729.318ns] or [-6.894%; +6.615%]

scenario:coreapiscenario-core_dispatch_1_listener

unstable execution_time [-28.541ns; +37.835ns] or [-4.668%; +6.188%]

scenario:coreapiscenario-core_dispatch_50_listeners

unstable execution_time [-1839.814ns; +1537.748ns] or [-10.675%; +8.922%]

scenario:coreapiscenario-core_dispatch_exception_listeners

unstable execution_time [-1278.442ns; +1187.769ns] or [-9.924%; +9.220%]

scenario:coreapiscenario-core_dispatch_listeners

unstable execution_time [-330.710ns; +323.345ns] or [-8.989%; +8.789%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

unstable execution_time [-234.703ns; +285.704ns] or [-8.011%; +9.752%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

unstable execution_time [-76.319ns; +70.179ns] or [-6.528%; +6.003%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

unstable execution_time [-3778.858ns; +4236.784ns] or [-9.270%; +10.394%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

unstable execution_time [-875.323ns; +685.747ns] or [-10.643%; +8.338%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

unstable execution_time [-57.425µs; +63.223µs] or [-9.006%; +9.915%]

brettlangdon

Some test coverage missing is _set_attributes re-entrancy, and should we get some tests for _set_default_attributes / set_default_attribute ?

Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>

cit-pr-commenter-54b7da · 2026-08-14T14:58:50Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

src/native/span/span_data.rs                                            @DataDog/apm-core-python
tests/tracer/test_span_tags.py                                          @DataDog/apm-sdk-capabilities-python

cit-pr-commenter-54b7da · 2026-08-14T14:59:06Z

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

cit-pr-commenter-54b7da · 2026-08-14T14:59:14Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 255 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 255 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.vllm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.profiling.scheduler -×-> ddtrace.trace  (product:profiling -> product:tracing, score=132)
ddtrace.internal.opentelemetry.trace -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=132)
ddtrace.internal.test_visibility.api -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

juanjux · 2026-08-14T15:39:56Z

Some test coverage missing is _set_attributes re-entrancy, and should we get some tests for _set_default_attributes / set_default_attribute ?

@brettlangdon agreed, done.

fix(tracing): avoid reentrant SpanData borrow panic

d8e1f22

Coerce Python span attributes before taking a mutable native borrow so lazy values can safely start nested spans.

juanjux requested review from a team as code owners August 14, 2026 12:27

juanjux requested review from emmettbutler and mabdinur August 14, 2026 12:27

chatgpt-codex-connector Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread src/native/span/span_data.rs Outdated

juanjux added the backport 4.14 label Aug 14, 2026

brettlangdon reviewed Aug 14, 2026

View reviewed changes

Comment thread tests/tracer/test_span_tags.py

Comment thread src/native/span/span_data.rs Outdated

Comment thread src/native/span/span_data.rs Outdated

juanjux and others added 2 commits August 14, 2026 16:57

Update src/native/span/span_data.rs

24941df

Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>

Update src/native/span/span_data.rs

3df0ecf

Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>

test(tracing): cover reentrant attribute mutation paths

fd0e7c7

juanjux requested a review from a team as a code owner August 14, 2026 15:36

brettlangdon approved these changes Aug 14, 2026

View reviewed changes

Comment thread releasenotes/notes/fix-reentrant-span-attribute-coercion-b92b8e914e27c9f4.yaml Outdated

juanjux added 2 commits August 14, 2026 17:57

chore: remove unnecessary release note

37b88a9

Merge branch 'main' into juanjux/fix-reentrant-span-data-borrow

8905c27

juanjux added the changelog/no-changelog A changelog entry is not required for this PR. label Aug 14, 2026

juanjux changed the title ~~fix(tracing): avoid reentrant SpanData borrow panic~~ chore(tracing): avoid reentrant SpanData borrow panic Aug 14, 2026

juanjux closed this Aug 14, 2026

juanjux reopened this Aug 14, 2026

juanjux enabled auto-merge (squash) August 14, 2026 16:14

Uh oh!

Conversation

juanjux commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional Notes

Uh oh!

datadog-prod-us1-3 Bot commented Aug 14, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

pr-commenter Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:httppropagationinject-ids_only

scenario:iastaspects-repr_aspect

scenario:iastaspects-upper_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:iastaspectssplit-rsplit_aspect

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

Uh oh!

brettlangdon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

juanjux commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

juanjux commented Aug 14, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Aug 14, 2026 •

edited by datadog-official Bot

Loading

pr-commenter Bot commented Aug 14, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading